Determining Yaw Error from Map Data, Lasers, and Cameras

ABSTRACT

The present disclosure relates to methods and systems that facilitate determination of a pose of a vehicle based on various combinations of map data and sensor data received from light detection and ranging (LIDAR) devices and/or camera devices. An example method includes receiving point cloud data from a (LIDAR) device and transforming the point cloud data to provide a top-down image. The method also includes comparing the top-down image to a reference image and determining, based on the comparison, a yaw error. An alternative method includes receiving camera image data from a camera and transforming the camera image data to provide a top-down image. The method also includes comparing the top-down image to a reference image and determining, based on the comparison, a yaw error.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Vehicles can be configured to operate in an autonomous mode in which thevehicle navigates through an environment with little or no input from adriver. Autonomous vehicles may include one or more sensors that areconfigured to detect information about the environment in which thevehicle operates.

Such sensors may include a light detection and ranging (LIDAR) device. ALIDAR can estimate distance to environmental features while scanningthrough a scene to assemble a “point cloud” indicative of reflectivesurfaces in the environment. Other sensors may include a visible lightcamera configured to capture image frames of the environment around thevehicle and/or a RADAR device configured to detect objects using radiofrequency signals.

SUMMARY

The present disclosure relates to methods and systems that facilitatedetermination of a pose of a vehicle based on various combinations ofmap data and sensor data received from light detection and ranging(LIDAR) devices and/or camera devices. For example, the presentdisclosure may provide systems and methods for determining andcorrecting a yaw error of a vehicle pose estimator.

In a first aspect, a method is provided. The method includes receivingpoint cloud data from a light detection and ranging (LIDAR) device. Themethod also includes transforming the point cloud data to provide atop-down image. The method further includes comparing the top-down imageto a reference image and determining, based on the comparison, a yawerror.

In a second aspect, a method is provided. The method includes receivingcamera image data from a camera and transforming the camera image datato provide a top-down image. The method also includes comparing thetop-down image to a reference image and determining, based on thecomparison, a yaw error.

In a third aspect, a method is provided. The method includes receivingreference data, wherein the reference data is provided from an overheadperspective. The method further includes transforming the reference dataso as to correspond to a field of view of a camera and receiving cameraimage data from the camera. The method additionally includes comparingthe transformed reference data to the camera image data and determining,based on the comparison, a yaw error.

Other aspects, embodiments, and implementations will become apparent tothose of ordinary skill in the art by reading the following detaileddescription, with reference where appropriate to the accompanyingdrawings.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a system, according to an example embodiment.

FIG. 2 illustrates a vehicle, according to an example embodiment.

FIG. 3A illustrates a top-down map image, according to an exampleembodiment.

FIG. 3B illustrates a top-down camera image, according to an exampleembodiment.

FIG. 3C illustrates a top-down aggregate image, according to an exampleembodiment.

FIG. 3D illustrates a top-down aggregate image, according to an exampleembodiment.

FIG. 4A illustrates a top-down map image, according to an exampleembodiment.

FIG. 4B illustrates a top-down LIDAR image, according to an exampleembodiment.

FIG. 4C illustrates a top-down aggregate image, according to an exampleembodiment.

FIG. 4D illustrates a top-down aggregate image, according to an exampleembodiment.

FIG. 5A illustrates a forward map image, according to an exampleembodiment.

FIG. 5B illustrates a forward camera image, according to an exampleembodiment.

FIG. 5C illustrates a forward aggregate image, according to an exampleembodiment.

FIG. 5D illustrates a forward aggregate image, according to an exampleembodiment.

FIG. 6 illustrates a method, according to an example embodiment.

FIG. 7 illustrates a method, according to an example embodiment.

FIG. 8 illustrates a method, according to an example embodiment.

DETAILED DESCRIPTION

Example methods, devices, and systems are described herein. It should beunderstood that the words “example” and “exemplary” are used herein tomean “serving as an example, instance, or illustration.” Any embodimentor feature described herein as being an “example” or “exemplary” is notnecessarily to be construed as preferred or advantageous over otherembodiments or features. Other embodiments can be utilized, and otherchanges can be made, without departing from the scope of the subjectmatter presented herein.

Thus, the example embodiments described herein are not meant to belimiting. Aspects of the present disclosure, as generally describedherein, and illustrated in the figures, can be arranged, substituted,combined, separated, and designed in a wide variety of differentconfigurations, all of which are contemplated herein.

Further, unless context suggests otherwise, the features illustrated ineach of the figures may be used in combination with one another. Thus,the figures should be generally viewed as component aspects of one ormore overall embodiments, with the understanding that not allillustrated features are necessary for each embodiment.

I. Overview

Conventional vehicle navigation systems may combine inertial sensors(e.g., accelerometers and gyroscopes) with aiding sensors (e.g., globalpositioning system (GPS), wheel speed sensors, camera, LIDAR, etc.)and/or supplementary information (e.g., maps) to determine a pose of thevehicle.

Systems and methods described herein provide accurate, frequentmeasurements of a pose of a vehicle, specifically its yaw, or heading.In many driving scenarios (e.g., while traveling on a relatively flatsurface), the yaw of the vehicle may include a compass orientation ofthe vehicle (e.g. north, south, east, west, etc).

Yaw measurements may be provided by GPS or by inferring yaw error basedon the lateral position error with respect to a map. However, it isdesirable to determine yaw errors directly. Described herein are systemsand methods that can determine small yaw errors directly from a map byusing LIDAR, radio detection and ranging (RADAR), and/or camerainformation.

In an example embodiment, point cloud data may be projected into atop-down image and compared to map data. Based on the comparison betweenthe projected point cloud data and the map data, the yaw error may bedirectly determined. For example, the yaw error may be determined as arotational angle adjustment that is suitable to register (e.g., align ormatch) the projected point cloud data correctly to the map data. Thedetermined yaw error may then be provided to the inertial navigationsystem as an adjustment signal.

In some other embodiments, camera image data may be projected into atop-down image and compared to map data. Based on the comparison betweenthe projected image data and the map data, the yaw error may be directlydetermined in a similar fashion as described above.

In further embodiments, map data may be projected into a perspectivecorresponding to the camera image frame. In such a scenario,left-to-right registration errors between the map data and camera imagesmay directly provide the yaw error.

In some embodiments, the map data may include reference data that couldbe generated from prior camera images, LIDAR point clouds, RADAR data,satellite imagery, other types of imagery, or other information aboutenvironment. Such information may be processed offline (e.g., by a cloudserver) and, in some embodiments, stored locally on the vehicle.

The yaw error may be inferred by various matching techniques, includingnormalized cross-correlation or other image registration methods. Insome embodiments, the matching could be performed on-board the vehicleor could be performed, at least in part, by a cloud server or anothertype of distributed computing system. In an example embodiment, theimage comparison and yaw error adjustment could be performed innear-real-time. Additionally or alternatively, the image comparisoncould be performed off-line to obtain information about, for example,long-term drift of a given sensor or reference data.

The systems and methods described herein may provide improved accuracyand reliability with regard to pose estimation for vehicles. Byproviding more accurate and reliable pose information, the overalllocalization of the vehicle may be better determined, which may lead tobetter perception, planning, obstacle avoidance, map-building,surveying, and change detection. Additionally or alternatively, theaccurate yaw information obtained with the methods and systems describedherein may be utilized with a variety of driving behaviors such as lanekeeping, obstacle avoidance, general navigation and routing, etc.Furthermore, in some cases, the methods and systems described herein mayprovide better yaw determination while using relatively low quality orlow accuracy sensors.

II. Example Systems

FIG. 1 illustrates a system 100, according to an example embodiment. Asshown, system 100 includes a vehicle 110, an overhead image source 130,and a controller 150.

Vehicle 110 may include a car, truck, aircraft, boat, mobile robot oranother type of moving object. The vehicle 110 may include varioussensors, such as one or more LIDAR systems 112, one or more cameras 114,and/or one or more RADAR systems 116. In some embodiments, vehicle 110may move in a semi- or fully-autonomous manner, based, at least in part,on information received from the various sensors and/or reference data118.

The LIDAR 112 may be configured to provide information (e.g., pointcloud data) about one or more objects (e.g., location, shape, etc.) inthe environment of the vehicle 110. For instance, the LIDAR 112 may emitlight pulses into the environment. The emitted light pulses may interactwith objects in the environment to form reflected light and the LIDAR112 may receive at least a portion of the reflected light as a reflectedsignal. For example, the LIDAR 112 may compare a time at which eachrespective light pulse is emitted with a time when each correspondingreflected light pulse is received and determine the distance between theone or more objects and the vehicle 110 based on the comparison.

In some embodiments, LIDAR 112 may be configured to rotate about an axis(e.g., a vertical axis of vehicle 110). In such scenarios, athree-dimensional map of a 360 degree view of the environment of thevehicle 110 can be determined without frequent recalibration of thearrangement of the various components of the LIDAR 110. Additionally oralternatively, LIDAR 112 can be configured to tilt its axis of rotationto control the portion of the environment within which the LIDAR mayprovide environmental data.

In some embodiments, point cloud data obtained by the LIDAR 112 may beflattened, edited, or otherwise adjusted so as to form a top-down imageof the environment of the vehicle 110. For example, at least a portionof the three-dimensional point cloud data may be ignored, culled, and/ortransformed so as to provide the top-down image.

The camera 114 may be configured to capture still or video images.Camera 114 may include an image sensor operable to provide informationindicative of a field of view of the camera 114. In some embodiments,camera 114 may include an array of charge-coupled devices (CCDs) and/oran array of complementary metal oxide semiconductor (CMOS) detectordevices. Other types of detectors are possible. Such detectors may beoperable in the visible light wavelengths and/or in other wavelengths(e.g., infrared). In an example embodiment, the camera 114 could be aforward-looking camera (e.g., have a field of view corresponding to aforward direction of vehicle 110).

Additionally or alternatively, the camera 114 could include a pluralityof cameras with respective fields of view to cover some or all of anenvironment around the vehicle 110. For example, the camera 114 mayinclude a plurality of cameras configured to capture a 360 degree imageof the surroundings of the vehicle 110. In such a scenario, each camera114 of the plurality of cameras may capture a respective image. Thecorresponding plurality of images may be stitched or otherwise combinedto form a top-down image of the environment around the vehicle 110. Thatis, the top-down image could be formed based on one or more projectivegeometry transforms applied to the plurality of images.

RADAR 116 could include a system configured to actively estimatedistances to environmental features by emitting radio signals anddetecting returning reflected signals. In such a scenario, distances toradio-reflective features can be determined according to the time delaybetween transmission and reception. The RADAR 116 can emit a signal thatvaries in frequency over time, such as a signal with a time-varyingfrequency ramp, and then relate the difference in frequency between theemitted signal and the reflected signal to a range estimate. Somesystems may also estimate relative motion of reflective objects based onDoppler frequency shifts in the received reflected signals.

In some embodiments, the RADAR 116 may include directional antennas,which may be used for directional transmission and/or reception of RADARsignals to associate each range estimate with a bearing. More generally,directional antennas can also be used to focus radiated energy on agiven field of view of interest. Combining the measured distances andthe directional information allows for the surrounding environmentfeatures to be mapped. The RADAR sensor can thus be used, for instance,by an autonomous vehicle control system to avoid obstacles indicated bythe sensor information.

Vehicle 110 may receive, store, and/or process reference data 118. Invarious embodiments, reference data 118 could include a map or otherinformation indicative of an environment of the vehicle 110. Forexample, reference data 118 may include map data and/or overheadimagery. For example, the reference data 118 may include overheadimagery from an aircraft or satellite.

Additionally or alternatively, the reference data 118 could includepreviously-captured camera images (e.g., captured by camera 114 oranother camera). Yet further, the reference data 118 could includepreviously-captured LIDAR point cloud data (e.g., captured by LIDAR 112or another LIDAR system), or previously-captured RADAR data (e.g.,captured by RADAR 116 or another RADAR system). The reference data 118may include information about an environment of the vehicle 110. Forexample, the reference data 118 may include information about roads,building, landmarks (e.g., geographic locations), and other physicalcharacteristics about the environment of the vehicle 110. In someembodiments, the reference data 118 could be updated in real time.Furthermore, in some embodiments, the reference data 118 may berequested and/or received from a map database, which may be storedlocally or remotely (e.g., on a cloud-based computing network). Othertypes of reference data 118 are possible and contemplated herein.

The system 100 also includes a pose estimator 120. The pose estimator120 may be configured to approximate, estimate, determine, calculate, ortake other actions to ascertain pose information about the vehicle 110.The pose information of the vehicle 110 could include, among otherpossibilities, a yaw value (or heading), a pitch value, a roll value, analtitude, and/or a location of the vehicle 110. In other words, the poseestimator 120 may be operable to determine a current orientation and/orlocation of the vehicle in one, two, and/or three-dimensions.

In some embodiments, the yaw value of the vehicle 110 may be provided inunits of degrees. For example, if the vehicle 110 is headed, or facing,north, the yaw value provided by the pose estimator 120 may be 0°. Southmay correspond to a yaw value of 180°, and east and west may correspondto yaw values of 90° and 270°, respectively. It will be understood thatother ways of expressing yaw values, either in degrees, decimal numbers,or other types of nomenclature are possible and contemplated herein.

In a similar manner, the pitch and/or roll values of the vehicle 110 maybe provided by the pose estimator 120 as a function of degrees oranother variable. The altitude of the vehicle 110 may be provided by thepose estimator 120 as a function of feet or meters, or another measureof absolute or relative height. The location of the vehicle 110 could beprovided by the pose estimator 120 in global position system coordinatesor in another type of absolute or relative coordinate system.

The pose estimator 120 may include a processor and a memory. In someembodiments, the pose estimator 120 could include a field-programmablegate array (FPGA) or an application-specific integrated circuit (ASIC).In some embodiments, the pose estimator 120 could be another type ofcomputing system. Furthermore, the pose estimator 120 could be disposedlocally (e.g., on-board) and/or remotely with regard to the vehicle 110(e.g., cloud computing networks). In some embodiments, some or all ofthe functions of the pose estimator 120 could be carried out by thecontroller 150, described elsewhere herein.

The vehicle 110 may additionally include a communication interface 122.The communication interface 122 may be configured to providecommunication between the vehicle 110 and other systems, such as thecontroller 150, the overhead image source 130, one or more computingnetworks, and/or other vehicles. In some embodiments, the communicationinterface 122 could provide a communication link between variouselements of the vehicle 110.

The communication interface 122 could be, for example, a systemconfigured to provide wired or wireless communication between one ormore other vehicles, sensors, or other entities, either directly or viaa communication network. To this end, the communication interface 122may include an antenna and a chipset for communicating with the othervehicles, sensors, servers, or other entities either directly or via thecommunication network. The chipset or communication interface 122 ingeneral may be arranged to communicate according to one or more types ofwireless communication (e.g., protocols) such as BLUETOOTH, BLUETOOTHLOW ENERGY (BLE), communication protocols described in IEEE 802.11(including any IEEE 802.11 revisions), cellular technology (such as GSM,CDMA, UMTS, EV-DO, WiMAX, or LTE), ZIGBEE, dedicated short rangecommunications (DSRC), and radio frequency identification (RFID)communications, among other possibilities. The communication interface122 may take other forms as well.

The vehicle 110 additionally includes a guidance system 124. Theguidance system 124 may include a GPS, an inertial measurement unit(IMU), a gyroscope, and/or another type of device configured to provideinformation indicative of a location or pose of the vehicle 110. The GPSmay be any sensor (e.g., location sensor) configured to estimate ageographic location of the vehicle 110. To this end, the GPS may includea transceiver configured to estimate a position of the vehicle 110 withrespect to the Earth. The GPS may take other forms as well. The IMU mayinclude a combination of sensors configured to sense position andorientation changes of the vehicle 110 based on inertial acceleration.In some embodiments, the combination of sensors may include, forexample, accelerometers and gyroscopes. Other combinations of sensorsare possible as well.

The guidance system 124 may include various navigation and pathingcapabilities, which may determine, at least in part, a driving path forthe vehicle 110. The guidance system 124 may additionally be configuredto update the driving path dynamically while the vehicle 110 is inoperation. In some embodiments, the guidance system 124 may beconfigured to incorporate data from a sensor fusion algorithm, the GPS,the LIDAR system 112, and one or more predetermined maps (e.g.,reference data 118) so as to determine the driving path for the vehicle110. Guidance system 124 may also include an obstacle avoidance system,which may be configured to identify, evaluate, and avoid or otherwisenegotiate obstacles in the environment in which the vehicle 110 islocated. The vehicle 110 may additionally or alternatively includecomponents other than those shown.

In some implementations, system 100 may include an overhead image source130. In an example embodiment, the overhead image source 130 may beconfigured to capture map data, images, or other types of information soas to provide an overhead view of at least a portion of the environmentaround the vehicle 110. In some scenarios, the vehicle 110 may receiveand store the map data from the overhead image source 130, which may bean aircraft (e.g., a drone mapping aircraft) or a satellite. As anexample, map data could be stored with the overhead image source 130, atthe vehicle 110, and/or elsewhere.

System 100 also includes a controller 150. The controller 150 includesone or more processors 152 and at least one memory 154. The controller150 could be located onboard the vehicle 110 or remotely (e.g., a cloudcomputing network). Additionally or alternatively, some portions of thecontroller 150 may be disposed on the vehicle 110 while other portionsof the controller 150 are disposed elsewhere.

The controller 150 may include an on-board computer, an externalcomputer, or a mobile computing platform, such as a smartphone, tabletdevice, personal computer, wearable device, etc. Additionally oralternatively, the controller 150 may include, or be connected to, aremotely-located computer system, such as a cloud server. In an exampleembodiment, the controller 150 may be configured to carry out some orall method blocks or steps described herein.

The processor 152 may include, for instance, an application-specificintegrated circuit (ASIC) or a field-programmable gate array (FPGA).Other types of processors, computers, or devices configured to carry outsoftware instructions are contemplated herein. The memory 154 mayinclude a non-transitory computer-readable medium, such as, but notlimited to, read-only memory (ROM), programmable read-only memory(PROM), erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), non-volatilerandom-access memory (e.g., flash memory), a solid state drive (SSD), ahard disk drive (HDD), a Compact Disc (CD), a Digital Video Disk (DVD),a digital tape, read/write (R/W) CDs, R/W DVDs, etc.

The controller 150 may be configured to carry out various sensor fusionalgorithms, functions, or tasks. For example, the controller 150 may beconfigured to carry out a sensor fusion algorithm and/or a computerprogram product storing an algorithm. In carrying out the sensor fusionalgorithm, the controller 150 may receive or otherwise accept data fromvarious sensors (e.g., LIDAR 112, camera 114, RADAR 116 and/or othersensors) as an input. The data may include, for example, datarepresenting information sensed at the various sensors of the vehicle'ssensor system. The sensor fusion algorithm may include, for example, aKalman filter, a Bayesian network, an algorithm configured to performsome of the functions of the methods herein, or another algorithm. Thesensor fusion algorithm may further be configured to provide variousassessments based on the data from the sensor system, including, forexample, evaluations of individual objects and/or features in theenvironment in which the vehicle 110 is located, evaluations ofparticular situations, and/or evaluations of possible impacts based onparticular situations. Other assessments are possible as well.

The controller 150 may include a computer vision system, which mayinclude a system configured to process and analyze images captured by atleast one of: the LIDAR 112, camera 114, and/or RADAR 116 in an effortto identify objects and/or features in the environment in which thevehicle 110 is located. Such objects or features may include, forexample, traffic signals, other vehicles, pedestrians, cyclists,landmarks, road signs, roadways, construction zones, and other types ofobjects or obstacles. To this end, the computer vision system may use anobject recognition algorithm, a Structure from Motion (SFM) algorithm,video tracking, or other computer vision techniques. In someembodiments, the computer vision system may additionally be configuredto map the environment, track objects, estimate the speed of objects,etc.

In some embodiments, the controller 150 may be configured to carry outvarious image processing and/or point cloud data processing functions.That is, controller 150 may be configured to provide top-down imagesbased on respective camera images, RADAR data, and/or point cloud data.

The processor(s) 152 may be configured to execute program instructionsstored in memory 154 so as to carry out operations. As such, controller150 may be configured to carry out any or all of the operationsdescribed herein.

For example, the controller 150 may be configured to cause one or moresensor systems of the vehicle 110 to provide camera images, point clouddata, and/or radar data, which may provide information about objects inthe environment of vehicle 110. In some embodiments, the controller 150may cause the one or more sensor systems to provide information about adesired portion or region of the environment of the vehicle 110.

In an example, the controller 150 may receive point cloud data from theLIDAR 112 and transform the point cloud data to provide a top-downimage. In an example embodiment, the top-down image may include aflattened two-dimensional image of the environment around the vehicle110.

The controller 150 could also compare the top-down image to a referenceimage and determine, based on the comparison, a yaw error. In such ascenario, the reference image includes at least a portion of map data(e.g., reference data 118). In an example embodiment, the referenceimage includes at least a portion of at least one of: one or more priorcamera images, a prior LIDAR point cloud, prior RADAR data, satelliteimagery, or other overhead imagery.

The comparison of the top-down image to the reference image may beperformed in various ways. For example, the comparison may be performedwith one or more of: normalized cross-correlation, grayscale matching,gradient matching, histogram matching, edge detection (e.g., Canny edgedetection), scale-invariant feature transform (SIFT), and/or speeded-uprobust features (SURF). Additionally or alternatively, in someembodiments, the comparison may include various image registration ormorphing algorithms.

The controller 150 could also provide an adjustment signal to poseestimator 120. The adjustment signal could be based, at least in part,on the determined yaw error.

In such a scenario, the pose estimator 120 may be configured todetermine a yaw orientation of the vehicle 110. Upon receiving theadjustment signal, the pose estimator 120 may update a current estimatedyaw orientation of the vehicle 110.

In some embodiments, the controller 150 could be configured to carry outother operations to correct a determined yaw value based on a comparisonbetween camera image data with map data. For example, the controller 150may receive camera image data from camera 114. In an example, the cameraimage data includes information indicative of an environment around thevehicle 110.

The controller 150 may transform the camera image data to provide atop-down image. The top-down image includes a flattened two-dimensionalimage of the environment around the vehicle. In some embodiments, thetransformation of the camera image data providing the top-down image mayinclude a geometric perspective transform algorithm or another type ofimage processing.

As in prior examples described herein, controller 150 could beconfigured to compare the top-down image to a reference image (e.g.,reference data 118) and determine, based on the comparison, a yaw errorand adjust the current yaw orientation of the vehicle 110.

Additionally or alternatively, the controller 150 may be configured tocompare camera data to transformed map data. For example, the controller150 may receive reference data 118. In such a scenario, the referencedata 118 may include an image provided from an overhead perspective. Thereference data 118 may then be transformed by controller 150 so as tocorrespond to a field of view of the camera. The controller 150 may alsobe configured to receive camera image data from the camera and comparethe camera image data to the transformed reference data. Based on thecomparison, the controller 150 may determine, based on the comparison, ayaw error.

FIG. 2 illustrates a vehicle 200, according to an example embodiment.Vehicle 200 may be similar or identical to vehicle 110 as illustratedand described with reference to FIG. 1. While FIG. 2 illustrates anautomobile (e.g., a car), it will be understood that other types ofvehicles are possible and contemplated herein. The vehicle 200 mayinclude one or more sensor systems 202, 204, 206, 208, and/or 210. Theone or more sensor systems 202, 204, 206, 208, and 210 could be similaror identical to sensors described with regard to system 100. Namely,sensor systems 202, 204, 206, 208, and 210 could include one or more of:LIDAR sensors (e.g., LIDAR 112), cameras (e.g., camera 114), and/orRADAR sensors (e.g., RADAR 116). In some examples, sensor system 202could include a LIDAR system, camera, and a RADAR sensor, however othervariations, combinations, and arrangements of sensors are possible.

In an example embodiment, one or more of the sensor systems 202, 204,206, 208, and 210 could include LIDAR 112, which may be configured torotate about an axis (e.g., the vertical axis 230) so as to illuminatean environment around the vehicle 200 with light pulses. In an exampleembodiment, sensor systems 202, 204, 206, 208, and 210 may be configuredto provide respective point cloud information that may relate tophysical objects within the environment of the vehicle 200. In suchscenarios, based on detecting various aspects of reflected light pulses(e.g., the elapsed time of flight, polarization, etc.), informationabout the environment may be determined. Additionally or alternatively,LIDAR 112 could include a LIDAR system configured to controllably scan arelatively narrow desired field of view so as to determine point cloudinformation in a desired portion of the environment around the vehicle.Such a LIDAR system may be configured to provide better point cloudresolution at longer distances.

Similarly to LIDAR 112, camera 114 and/or RADAR 116 could be configuredto move in a desired manner with respect to the vehicle 200 so as toobtain dynamic information about the environment of the vehicle 200. Forexample, camera 114 and/or RADAR 116 may rotate around an axis or couldbe controllably oriented toward a desired field of view so as to provideinformation about a desired portion of the environment of the vehicle200.

While FIG. 2 illustrates sensor systems 202, 204, 206, 208, and 210 asbeing located at certain locations on the vehicle 200, it will beunderstood that many other arrangements of the sensors are possible andcontemplated.

A pose of the vehicle 200 may be provided in terms of yaw, roll, andpitch. For example, a yaw of vehicle 200 may be provided in terms of anangle 220, the roll of vehicle 200 may be provided in terms of angle240, and a pitch of vehicle 200 may be provided in terms of angle 222.Other ways to describe the pose of vehicle 200 are possible andcontemplated herein.

Several scenarios relating to the correction of an estimated yaw of avehicle will now be described. While the described scenarios relate toautonomous or semi-autonomous automobiles, it will be understood thatthe techniques, processes, and methods described herein may be appliedto many other types of vehicles as they move within a particularenvironment.

Self-driving vehicles typically use yaw (heading) information to performmany different driving behaviors (e.g., lane keeping, obstacleavoidance, general navigation and routing, etc.). Thus, better yawestimation can improve with route planning, map building, surveying, andchange detection. The scenarios described below may provide ways torefine yaw estimates from relatively inexpensive sensor systems.

While embodiments described herein may relate to heading (yaw), it isunderstood that other refinements to estimated pose are possible bycomparing various reference data and sensor data. For example, anestimated pitch or roll of a vehicle may be refined using methods orsystems described herein.

FIGS. 3A-3D illustrate various images according to example embodiments.

FIG. 3A illustrates a top-down map image 300, according to an exampleembodiment. Top-down map image 300 may be a portion of the referencedata 118 as illustrated and described in reference to FIG. 1. Thetop-down map image 300 may include information about various features ofan environment (e.g., an environment around a vehicle). Such featurescould include, without limitation, roadways 302, sidewalks 304, lanemarkers 306, curbs/medians 308, and service covers 310 (e.g., manholesor drains). In some embodiments, the top-down map image 300 may bepreprocessed to remove temporally varying objects (e.g., movingvehicles, people, bicyclists, intervening cloud cover/haze, etc.) orotherwise standardize/normalize the image.

In some embodiments, top-down map image 300 may be provided as an imagecaptured by overhead image source 130.

FIG. 3B illustrates a top-down camera image 320, according to an exampleembodiment. The top-down camera image 320 may be similar or identical toone or more images provided by camera 114, as illustrated and describedin reference to FIG. 1. In some embodiments, the top-down camera image320 may include several camera images that have been stitched togetherand adjusted so that it looks like a top-down image, using, for example,a perspective morphing technique.

Top-down camera image 320 may include, for example, some features thatare similar or identical to those of FIG. 3A. For example, in an exampleembodiment, the top-down camera image 320 may include roadways 302,sidewalks 304, lane markers 306, curbs/medians 308, and/or servicecovers 310. Furthermore, the top-down camera image 320 may includereal-time information about objects in the environment of a vehicle 200.As an example, top-down camera image 320 may include other vehicles 322and a bicyclist 324.

In some embodiments, the top-down camera image 320 may be adjusted byremoving or ignoring temporally varying objects, such as the othervehicles 322 and bicyclist 324. Other objects that are recognized asbeing likely to change over time (e.g., pedestrians, cloud cover, etc.)may be deleted, ignored, or otherwise handled outside of the methodsdescribed elsewhere herein.

FIG. 3C illustrates a top-down aggregate image 340, according to anexample embodiment. Top-down aggregate image 340 may include acombination, overlay, superposition, juxtaposition, comparison, oranother type of analysis of the top-down map image 310 and the top-downcamera image 320. As illustrated in FIG. 3C, an embodiment may includethe aggregate image 340 being an overlay of the two images. In such ascenario, the top-down aggregate image 340 may include an angledifference 342. The angle difference 342 may include a differencebetween an estimated heading and an actual heading. In some cases, theangle difference 342 may be provided to the pose estimator 120 and/orthe guidance system 124 to improve the estimated heading of the vehicle.

FIG. 3D illustrates a top-down aggregate image 350, according to anexample embodiment. Aggregate image 350 may illustrate a combination ofmap image 300 and camera image 320. Namely, areas in which the cameraimage 320 corresponds with the map image 300 may be drawn into theaggregate image 350. In some embodiments, aggregate image 350 couldinclude portions without substantive image information. Such portionsmay correspond to areas that have not been imaged by the camera and/orportions of the image that are substantially different between the mapimage and the camera image.

FIG. 4A illustrates a top-down map image 400, according to an exampleembodiment. Top-down map image 400 may be formed from reference data118, as illustrated and described in reference to FIG. 1. Top-down mapimage 400 may include, without limitation, many different types ofobjects in the environment of a vehicle. For example, top-down map image400 may include roadways 402, lane lines 403, sidewalks 404, and trees406. In some embodiments, the top-down map image 400 may be preprocessedto remove temporally varying objects (e.g., moving vehicles, people,bicyclists, exhaust, etc.) or otherwise standardize/normalize the image.In example embodiments, vegetation/foliage may be partially or fullyremoved from, or ignored within, the top-down map image 400. In someembodiments, top-down map image 400 may be provided as an image capturedby overhead image source 130.

FIG. 4B illustrates a top-down LIDAR image 420, according to an exampleembodiment. The top-down LIDAR image 420 may include at least some ofthe features described with reference to FIG. 4A. Furthermore, the LIDARimage 420 may include other vehicles 422 and/or other time varyingelements. The top-down LIDAR image 420 may include processed data frompoint cloud data provided by LIDAR 112 or another LIDAR system. Forexample, LIDAR 112 may provide point cloud information to controller 150or another type of imaging processing computer system. In such ascenario, the three-dimensional point cloud information may becompressed/flattened/or otherwise adjusted for perspective to form atwo-dimensional top-down perspective image 420.

FIG. 4C illustrates a top-down aggregate image 440, according to anexample embodiment. The top-down aggregate image 440 may illustrate acombination, overlay, superposition, or another type of image comparisonbetween the top-down map image 400 and the top-down LIDAR image 420. Thetop-down aggregate image 440 may include some of the same features asthose in FIGS. 4A and 4B. For example, image 440 may include roadways402, lane lines 403, sidewalks 404, and trees 406. In some embodiments,the top-down aggregate image 440 could, but need not, include the othervehicles 422 and bicyclist 424. In some embodiments, the comparisonbetween the map image 400 and the LIDAR image 420 may ignore, delete,average, or otherwise discount or deprioritize objects recognized to betemporally changing or impermanent.

FIG. 4D illustrates a top-down aggregate image 450, according to anexample embodiment. The non-black areas of aggregate image 450 mayindicate features from LIDAR image 420 that matches, corresponds, orotherwise similar to that of map image 400. In another embodiment, atleast some of the black areas of aggregate image 450 may representunscanned areas (e.g., no point cloud data) due to such areas beingoccluded or blocked from the LIDAR's scanning view.

FIG. 5A illustrates a forward map image 500, according to an exampleembodiment. The forward map image 500 may include a top-down map image(e.g., map images 300 and 400) that has been adjusted, manipulated, orotherwise changed so as to have a similar or identical perspective asthat of a forward-looking camera (e.g., camera 114). It will beunderstood that while a forward-looking camera view is contemplated inembodiments described herein, methods and systems involving otherviewpoints and/or perspectives are possible.

The forward map image 500 may include roadways 502, lane lines 503,traffic signs/lights 504, trees 506, buildings, landmarks, etc. Otherfeatures that may be provided in a map image are possible andcontemplated herein.

FIG. 5B illustrates a forward camera image 520, according to an exampleembodiment. Forward camera image 520 may include similar features tothose of forward map image 500. For example, forward camera image 520could include roadways 502, lane lines 503, traffic signs/lights 504,trees 506, and various other vehicles 522 and 524. In some embodiments,the forward camera image 520 may be captured by a video or still camera(e.g., camera 114). While FIG. 5B includes a forward camera image 520,it is understood that LIDAR point cloud data could transformed so as toform a forward LIDAR “image” with a forward-looking perspective.

FIG. 5C illustrates a forward aggregate image 540, according to anexample embodiment. The forward aggregate image 540 may illustrate acombination, overlay, superposition, or another type of image comparisonbetween the forward map image 500 and the forward camera image 520 (orforward LIDAR image). The forward aggregate image 540 may include somefeatures illustrated in FIGS. 5A and 5B. For example, image 540 mayinclude roadways 502, lane lines 503, and trees 506. In someembodiments, the forward aggregate image 540 could, but need not,include the other vehicles 522 and 524. In some embodiments, thecomparison between the forward map image 500 and the forward cameraimage 520 may ignore, delete, average, or otherwise discount objectsrecognized to be temporally changing or impermanent. In someembodiments, fiducial images 542 could be superimposed onto or comparedto lane lines 503 or other characteristic features or landmarks in image540. In some cases, fiducial images 542 may be representative of LIDAR,RADAR, and/or map data.

In some cases, the forward aggregate image 540 may suggest that anactual yaw direction may be different than that of an estimated yawdirection. In such cases, the angle difference may be determined basedon an image analysis. Also, an adjusted yaw signal may be provided tocontroller 150, pose estimator 120, guidance system 124 or anothercontrol system so as to refine the actual yaw position of the vehicle.

FIG. 5D illustrates a forward aggregate image 550, according to anexample embodiment. The dotted shapes near the broken lane lines mayinclude a graphical representation of LIDAR point cloud data, which maybe similar or identical to the fiducial images 542 as illustrated anddescribed in reference to FIG. 5C.

III. Example Methods

FIG. 6 illustrates a method 600, according to an example embodiment.Method 600 may be carried out, in full or in part, by vehicle 110,controller 150, or vehicle 200 as illustrated and described in referenceto FIGS. 1 and 2. Method 600 may include elements that are similar oridentical to those illustrated and described with reference to FIGS. 1,2, 3A-3D, 4A-4D, and/or 5A-5D. It will be understood that the method 600may include fewer or more steps or blocks than those expressly disclosedherein. Furthermore, respective steps or blocks of method 600 may beperformed in any order and each step or block may be performed one ormore times. In some embodiments, method 600 may be combined with one ormore of methods 700 or 800 as illustrated and described in reference toFIGS. 7 and 8.

Block 602 includes receiving point cloud data from a light detection andranging (LIDAR) device. In some embodiments, the point cloud dataincludes information indicative of objects in an environment around avehicle.

Block 604 includes transforming the point cloud data to provide atop-down image. In such a scenario, the top-down image includes aflattened two-dimensional image of the environment around the vehicle.For example, the transforming could include a scale-invariant featuretransform, image flattening, image perspective rotation, and/or a3D-to-2D image transform. Other ways to represent three-dimensionalobject information as a two-dimensional, top-down image are contemplatedherein.

Block 606 includes comparing the top-down image to a reference image,which may include at least a portion of map data. The reference imagemay include at least a portion of at least one of: one or more priorcamera images, a LIDAR point cloud, RADAR data, satellite imagery, othertypes of imagery, or other types of information about the environment.

Block 608 includes determining, based on the comparison, a yaw error.

Additionally or optionally, method 600 may include providing anadjustment signal to a pose estimator of a vehicle based on thedetermined yaw error. In such scenarios, the method 600 may additionallyinclude determining, with the inertial pose estimator, a yaw orientationof the vehicle and determining a refined yaw orientation based on theadjustment signal and the determined yaw orientation.

FIG. 7 illustrates a method 700, according to an example embodiment.Method 700 may be carried out, in full or in part, by vehicle 110,controller 150, or vehicle 200 as illustrated and described in referenceto FIGS. 1 and 2. Method 700 may include elements that are similar oridentical to those illustrated and described with reference to FIGS. 1,2, 3A-3D, 4A-4D, and/or 5A-5D. It will be understood that the method 700may include fewer or more steps or blocks than those expressly disclosedherein. Furthermore, respective steps or blocks of method 700 may beperformed in any order and each step or block may be performed one ormore times. In some embodiments, method 700 may be combined with one ormore of methods 600 or 800 as illustrated and described in reference toFIGS. 6 and 8.

Block 702 includes receiving camera image data from a camera. Asdescribed elsewhere herein, the camera image data may includeinformation indicative of an environment around a vehicle. For example,the camera image data may include information about a vehicle roadway,driving path, obstacles, landmarks, and/or other objects in thevehicle's environment.

Block 704 includes transforming the camera image data to provide atop-down image. In an example embodiment, the top-down image may includea flattened two-dimensional image of the environment around the vehicle.

Block 706 includes comparing the top-down image to a reference image. Insome embodiments, the reference image includes at least a portion of mapdata. For example, the reference image may include at least a portion ofat least one of: one or more prior camera images, a LIDAR point cloud,RADAR data, satellite imagery, other types of imagery, or otherinformation about the environment.

Block 708 includes determining, based on the comparison, a yaw error.

The method 700 may also include providing an adjustment signal to a poseestimator of a vehicle based on the determined yaw error.

In an example embodiment, the method 700 may include determining, e.g.,with the pose estimator, a yaw orientation of the vehicle. Based on theadjustment signal, the method 700 may also include determining a refinedyaw orientation based on the adjustment signal and the determined yaworientation.

FIG. 8 illustrates a method 800, according to an example embodiment.Method 800 may be carried out, in full or in part, by vehicle 110,controller 150, or vehicle 200 as illustrated and described in referenceto FIGS. 1 and 2. Method 800 may include elements that are similar oridentical to those illustrated and described with reference to FIGS. 1,2, 3A-3D, 4A-4D, and/or 5A-5D. It will be understood that the method 800may include fewer or more steps or blocks than those expressly disclosedherein. Furthermore, respective steps or blocks of method 800 may beperformed in any order and each step or block may be performed one ormore times. In some embodiments, method 800 may be combined with one ormore of methods 600 or 700 as illustrated and described in reference toFIGS. 6 and 8.

Block 802 includes receiving reference data. In such a scenario, thereference data may be information indicative of an overhead perspectiveof a scene. In some embodiments, the reference data includes at least aportion of at least one of: one or more prior camera images, a LIDARpoint cloud, RADAR data, satellite imagery, other types of imagery, orother information about the environment.

Block 804 includes transforming the reference data so as to correspondto a field of view of a camera. The transforming could include, withoutlimitation, 3D-to-2D image transform, a scale-invariant featuretransform, or another type of image processing transform configured toadjust a perspective of image data, point cloud data, or other types ofenvironmental data. In some embodiments, the transforming could changereference data from an overhead perspective to a forward-facingperspective. Other types of image processing transforms are possible andcontemplated herein.

Block 806 includes receiving camera image data from the camera. In anexample embodiment, the camera image data may include a flattenedtwo-dimensional image of at least a portion of the environment aroundthe vehicle.

Block 808 includes comparing the transformed reference data to thecamera image data. In some embodiments, the camera image datacorresponds to at least a portion of the transformed reference data. Insome scenarios, the reference data and/or the camera image data may beprocessed with at least one of: an image difference algorithm, an imagehomography algorithm, or an image rectification algorithm.

Block 810 includes determining, based on the comparison, a yaw error. Insome embodiments, method 800 further includes providing an adjustmentsignal to a pose estimator of a vehicle based on the determined yawerror.

The particular arrangements shown in the Figures should not be viewed aslimiting. It should be understood that other embodiments may includemore or less of each element shown in a given Figure. Further, some ofthe illustrated elements may be combined or omitted. Yet further, anillustrative embodiment may include elements that are not illustrated inthe Figures.

A step or block that represents a processing of information cancorrespond to circuitry that can be configured to perform the specificlogical functions of a herein-described method or technique.Alternatively or additionally, a step or block that represents aprocessing of information can correspond to a module, a segment, aphysical computer (e.g., a field programmable gate array (FPGA) orapplication-specific integrated circuit (ASIC)), or a portion of programcode (including related data). The program code can include one or moreinstructions executable by a processor for implementing specific logicalfunctions or actions in the method or technique. The program code and/orrelated data can be stored on any type of computer readable medium suchas a storage device including a disk, hard drive, or other storagemedium.

The computer readable medium can also include non-transitory computerreadable media such as computer-readable media that store data for shortperiods of time like register memory, processor cache, and random accessmemory (RAM). The computer readable media can also includenon-transitory computer readable media that store program code and/ordata for longer periods of time. Thus, the computer readable media mayinclude secondary or persistent long term storage, like read only memory(ROM), optical or magnetic disks, compact-disc read only memory(CD-ROM), for example. The computer readable media can also be any othervolatile or non-volatile storage systems. A computer readable medium canbe considered a computer readable storage medium, for example, or atangible storage device.

While various examples and embodiments have been disclosed, otherexamples and embodiments will be apparent to those skilled in the art.The various disclosed examples and embodiments are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

What is claimed is:
 1. A method comprising: receiving point cloud datafrom a light detection and ranging (LIDAR) device; transforming thepoint cloud data to provide a top-down image; comparing the top-downimage to a reference image; determining, based on the comparison, a yawerror.
 2. The method of claim 1, wherein the point cloud data comprisesinformation indicative of objects in an environment around a vehicle. 3.The method of claim 2, wherein the top-down image comprises a flattenedtwo-dimensional image of the environment around the vehicle.
 4. Themethod of claim 1, further comprising: providing an adjustment signal toa pose estimator of a vehicle based on the determined yaw error.
 5. Themethod of claim 4, further comprising: determining, with the poseestimator, a yaw orientation of the vehicle; determining a refined yaworientation based on the adjustment signal and the determined yaworientation.
 6. The method of claim 1, wherein the reference imagecomprises at least a portion of map data.
 7. The method of claim 1,wherein the reference image comprises at least a portion of at least oneof: one or more prior camera images, a LIDAR point cloud, RADAR data,satellite imagery, or other overhead imagery.
 8. A method comprising:receiving camera image data from a camera; transforming the camera imagedata to provide a top-down image; comparing the top-down image to areference image; determining, based on the comparison, a yaw error. 9.The method of claim 8, wherein the camera image data comprisesinformation indicative of an environment around a vehicle.
 10. Themethod of claim 8, wherein the top-down image comprises a flattenedtwo-dimensional image of an environment around a vehicle.
 11. The methodof claim 8, further comprising: providing an adjustment signal to a poseestimator of a vehicle based on the determined yaw error.
 12. The methodof claim 11, further comprising: determining, with the pose estimator, ayaw orientation of the vehicle; determining a refined yaw orientationbased on the adjustment signal and the determined yaw orientation. 13.The method of claim 8, wherein the reference image comprises at least aportion of map data.
 14. The method of claim 8, wherein the referenceimage comprises at least a portion of at least one of: one or more priorcamera images, a LIDAR point cloud, RADAR data, satellite imagery, orother overhead imagery.
 15. A method comprising: receiving referencedata, wherein the reference data comprises information indicative of anoverhead perspective of a scene; transforming the reference data so asto correspond to a field of view of a camera; receiving camera imagedata from the camera; comparing the transformed reference data to thecamera image data; determining, based on the comparison, a yaw error.16. The method of claim 15, wherein the camera image data comprises aflattened two-dimensional image of an environment around a vehicle. 17.The method of claim 15, further comprising: providing an adjustmentsignal to a pose estimator of a vehicle based on the determined yawerror.
 18. The method of claim 15, wherein the camera image datacorresponds to at least a portion of the transformed reference data. 19.The method of claim 15, wherein the reference data comprises at least aportion of at least one of: one or more prior camera images, a LIDARpoint cloud, RADAR data, satellite imagery, or other overhead imagery.20. The method of claim 15, wherein at least one of: the reference dataor the camera image data are processed with at least one of: an imagedifference algorithm, an image homography algorithm, or an imagerectification algorithm.